trentmkelly/LessWrong-43k huggingface dataset:
The low cost of human preference incoherence
by Stuart_Armstrong — 2019-03-27T11:58:14.845Z

Note: working on a research agenda, hence the large amount of small individual posts, to have things to link to in the main documents.

In aiming for an adequate synthesis of human preferences, I tend to preserve quirks and even biases of humans, rather than aiming for elegant simplicity as many philosophers do. The main reason is that, feeling that value is fragile, I fear losing a valid preference more than allowing an illegitimate bias to get promoted to preference.

But another point is that I don't feel that the cost of excessive complexity is very high.

The small cost of one-off restrictions

An analogy could be with government interventions in the market: most naive restrictions will only slightly raise the cost, but will not have a large effect, as the market routes around the restrictions. Consider a government that created price controls on all bread, for some reason. If that was a one-off rule, then we know what would happen: loaves of bread would get gradually smaller, while bakeries would start producing more cake or cake-like products, which now occupy the niche that more expensive bread products would otherwise have filled.

There is likely an efficiency loss here - the balance between cake and bread is different from the market would have given, for small losses to bakers and customers. But these are not huge losses, and they are certainly much smaller than would have happened if the bakers and customers had simply continued as before, except with the price control.

Note that if the government is dynamic with its regulations, then it can impose effective price controls. It could add regulations about bread size, then bread quality, repeatedly; it could apply price controls to cake as well or ban cakes; as the market adjusts, regulations could adjust too. This would impose large efficiency losses (possibly coupled with equity gains, in certain circumstances).

But the cost of one-off restrictions tends to be low.

Quirks and biases in human preferences

It seems to me that including quirks and biases in human preferences, is akin to one-off restrictions in the market. Firstly because these quirks will be weighted and balanced against more standard preferences, so would be weaker than "government regulations". They would be one-off: quirks and biases tend to be, by definition, less coherent than standard preferences, so they wouldn't be particularly consis

...
Dependability
by Unreal — 2019-03-26T22:49:37.402Z

I have become very, very interested in developing a skill that I call Dependability.

I believe the skill exists on a spectrum, and you can have less or more of it. It’s not a binary where you either have it or you don’t.

I believe this skill can be trained on purpose.

I will briefly describe each attribute that makes up the overall skill.

( All the examples are made up. Also, assume that the examples are only talking about endorsed actions and goals. )

Trying

To have a vision of a skill or a desirable end state and then be able to strive with deliberate effort towards making that vision a reality.

Ex1: I see a dance routine on YouTube that I think would be awesome if I could perform myself. I’ve never done anything like this before, and I’m somewhat self-conscious or skeptical of how likely I am to succeed. Regardless, I take concrete steps towards learning it (study the video, practice the moves that I see, repeat for many days until I’ve achieved competency at performing the dance). There is some possibility I fail for whatever reason, but this doesn’t stop me from giving it my full effort for at least a week.

Ex2: There’s a job that I really want. I’m unclear about what steps I need to take to acquire the job, and I’m not sure I’m qualified. I research what kinds of skills and traits are desirable in the job by asking people, Googling, and looking through applications. (I am more encouraged than not by this initial research.) I sign up for workshops and classes that will give me relevant training. I read books. I practice in my free time. I make useful connections / network. I build whatever reputational capital seems useful via blogging, social media, in-person meetings, running events. I apply for the job. If I fail, I figure out what needs work, fix it, and try again until I obtain the position.

Commitment

To form an intention to do something (generally on a longer time scale), be able to say it out loud to someone else, and then be certain it will happen one way or another, barring extreme circumstance.

Ex1: I commit myself via marriage to another person and promise that I will try everything to make the relationship work before giving up on it. I say it out loud as a vow to the other person in a marriage ceremony, in front of a bunch of people. Then I proceed to actually attempt to get as close to 100% chance of creating a permanent relationship situatio...
Unsolved research problems vs. real-world threat models
by catherio — 2019-03-26T22:10:08.371Z
Link: https://medium.com/@catherio/unsolved-research-problems-vs-real-world-threat-models-e270e256bc9e

I've given a lightning talk twice now about "why should we care about adversarial examples?", highlighting the distinction (which ML experts are usually aware of, but media and policy folks often miss) between an unsolved research problem and a real-world threat model.

I've now published a version of this as a general-audience Medium post.

For a LessWrong / Alignment Forum audience, I'll highlight the fact that I'm directly and consciously drawing on the "intellectual puzzle" vs. "instrumental strategy" framing from Abram Demski's Embedded Curiosities post (at the conclusion of the Embedded Agency series).

I'm mostly making an observation at the level of "how a research field works", and find it interesting that agent foundations and adversarial examples research are both hitting similar confusions when justifying the importance of their work.

I also think a lot more could be said and explored on the object-level about which angles on "adversarial examples" are likely to yield valuable insights in the long run, and which are dead ends. But this post isn't really about that.
A Concrete Proposal for Adversarial IDA
by evhub — 2019-03-26T19:50:34.869Z

Note: This post came out of a conversation with Geoffrey Irving and Buck Shlegeris.

Epistemic Status: I suspect Paul has already thought of most or all of the ideas presented here, though I nevertheless found the exercise of carefully specifying an IDA implementation helpful and suspect others may find reading it helpful as well.

This is a proposal for how to train a machine learning model to approximate HCH using Iterated Distillation and Amplification (IDA). This particular proposal came out of a desire to use a debate-like adversary to improve the amplification process, and the primary goal of this proposal is to show how one could do that. Though I have tried to retain a lot of the relevant detail, I have made two simplifications to make this proposal easier to specify: I am attempting to approximate something closer to weak HCH rather than strong HCH and I am only allowing the generation of two subquestions at a time. I am confident that those simplifications could easily be dropped, though I think doing so here would only make this presentation more complicated.

Before I proceed, I want to make one final note: this is not a proposal for how to build an aligned AGI. I think there are still a whole bunch of issues that would prevent this proposal from actually working.

Definitions

We will start with some initial definitions:

Let Q be the set of all questions in natural language.
Let A be the set of all answers in natural language.
Let M be the sum type of either Q×Q or A representing either an answer to the given question or two subquestions to help answer it.
Let H:Q→A be the answer that a human gives to the given question.
Let Hfan out:Q→M be the answer or subquestion pair generated by a human when asked what to do with the given question.
Let Hfan in:Q×(Q×A)×(Q×A)→M be the answer or two subquestions generated by a human to some question when given answers to two subquestions related to that question.
Let ML:Q→Δ(A) be a model (the training procedure for which we will describe below) from questions to a probability distribution over strings representing answers. Specifically, we will implement the probability distribution by having our model output an embedding vector which yields the probability distribution when fed into some trained language model (by repeatedly conditioning on previous characters and mu

...
Deck Guide: Biomancer’s Familiar
by Zvi — 2019-03-26T15:20:00.420Z

Biomancer’s Familiar is a great card. I wanted it to happen so bad. I spent a substantial portion of my preperations for Cleveland trying to make Biomancer’s Familiar happen. I tried two color versions. I tried three color versions. I tried going big, going small, going wide, and everything else I could think of. I came close enough to consider buying the cards.

Ultimately, I could not make it happen. Blue was too strong and too structurally tough. You had to give up Pelt Collector, had to pay real mana for your spells. The format wanted different things than the Familiar deck could provide. Sideboarding wasn’t as impactful for you as I wanted. The deck was good. It was fun as hell. But not good enough. I switched to blue, tore up the ladder with it, and never looked back. Things didn’t work out at the Pro Tour, but given the overall results, I am confident I made the right decision.

There are three reasons to share the deck now.

The first reason is, as noted, that the deck is great fun. As constructed, it’s a step behind where it needs to be to win major tournamens, but it’s still good enough to get five wins more often than not in Traditional Constructed. Its best draws bury people if not answered, and it gets them often.

The second reason is that perhaps I missed something. There’s a lot of good things going on, so the deck might be one card, or one idea or sideboard plan, away from competitive. That might be a two color or three color build. It might involve a card from War of the Spark.

The third reason is that now that I write it out, I think this deck plays fine against what’s out there right now. My results weren’t as good as with blue, but perhaps things have changed.

Here is the strongest version of the deck:

3 Adventurous Impulse

2 Quench

2 Negate

4 Llanowar Elves

4 Biomancer’s Familiar

4 Growth-Chamber Guardian

4 Incubation Druid

2 Druid of the Cowl

4 Sphinx of Foresight

4 Frilled Mystic

4 Biogenic Ooze

10 Forest

5 Island

4 Hinterland Harbor

4 Breeding Pool

Sideboard

4 Kraul Harpooner

1 Thorn Lieutenant

4 Entrancing Melody

2 Negate

2 Spell Pierce

2 Dive Down

There are only eight adapt creatures in the deck for Biomancer’s Familiar. This seems light, but you have a lot of search, and more of that is not what you need. There are places where a Xengara, Utopia Speaker or two would be welcome for additional power, but I found that if we were going large and finding ways to clo...
[Method] The light side of motivation: positive feedback-loop 
by Vincent B — 2019-03-26T14:05:12.533Z

I want to share this method I use sometimes to stay focused on my tasks, earn rewards from them, and build up a positive feedback-loop to do more difficult things. It's nothing new and has probably been written about a few times. I have been using it subconsciously for years, and wanted to do an explicit representation for future use. If this sounds completely wrong to you, please ignore it or tell me in the comments.

It should go without mentioning that this is just one part of a well-tuned system. It works because other parts work and support it. If supportive systems are wired differently or broken, this approach may not work at all.

What you need

the ability to motivate yourself to some degree
some preparation, or else that you have a mental list of tasks that can be done
the ability to complete simple or moderately difficult tasks when you are already motivated (Side-note: Forcing yourself to do things when you really, really don't want to might be effective one or twice. But in the long run, it's going to build up an even stronger aversion. Then making yourself do the thing will be even more difficult. How to change your feelings about a task is not the main topic of this post.)

How it works

Step 1: Induce happiness. Make yourself feel confident and hopeful.
Step 2: Choose the (simple to moderately difficult) action you want to complete.
Step 3: Complete the action and earn reward for this! With the "evidence" of being able to complete actions, stir up your confidence of being able to complete tasks.
Step 4: Choose the next stack of actions. They should be moderately difficult and involve clear steps to the solution (no 'meta-actions'). Begin them immediately, before your confidence and motivation fades; in doing so, focus on the short-term future when you are going to feel accomplished about having completed them. Use your recently activated confidence for this. Don't focus on your potential dislike of them or any other feelings of avoidance. If they crop up regardless, ignore them and tell them that they are going to be defeated soon, so they should just leave. (I know how this sounds. But if you are anything like me, treating your mind like a dog to be trained can be really helpful in getting it to do things!)
Step 5: Relax from having successfully completed a stack of necessary and useful actions. Bask in the reward,

...
"Moral" as a preference label
by Stuart_Armstrong — 2019-03-26T10:30:17.102Z

Note: working on a research agenda, hence the large amount of small individual posts, to have things to link to in the main documents.

In my quest to synthesise human preferences, I've occasionally been asked whether I distinguish moral preferences from other types of preferences - for example, whether preferences for Abba or Beethoven, or avocado or sausages, should rank as high as human rights or freedom of speech.

The answer is, of course not. But these are not the sort of things that should be built into the system by hand. This should be reflected in the meta-preferences. We label certain preferences "moral", and we often have the belief that these should have priority, to some extent, over merely "selfish" preferences (the extent of this belief varies from person to person, of course).

I deliberately wrote the wrong word there for this formalism - we don't have the "belief" that moral preferences are more important, we have the meta-preference that a certain class of beliefs, labelled "moral", whatever that turns out to mean, should be given greater weight. This is especially the case as there are a lot of cases where it is very unclear if a preference is moral or not (many people have strong moral-ish preferences over mainstream cultural and entertainment choices).

This is an example of the sort of challenges that a preference synthesis process should be able to figure out on its own. If the method needs to be constantly tweaked to get over every small problem of definition, then it cannot work. As always, however, it need not get everything exactly right; indeed, it needs to be robust enough that it doesn't change much if a borderline meta-preference such as "everyone should know their own history" gets labelled as moral or not.
What I've Learned From My Parents' Arranged Marriage
by squidious — 2019-03-26T06:40:00.975Z

When I tell people my parents had an arranged marriage, I get a number of different reactions. Most people have the wrong idea of exactly what that looks like, and those who do have the right idea often wonder if my parents can even understand what dating is like, given they've never experienced it. I've heard people assume that my parents' arranged marriage meant they were completely unable to help or give advice when it came to my dating life, and I've found the opposite to be the case; the advice my parents gave me about dating was as valuable as anything I found anywhere else, and allowed me to pass that advice on to my friends. Growing up hearing their story taught me a lot about what was important to know about myself before I started dating anyone, and how a good couple functions and grows together. I found that much of this is less commonly talked about when it comes to Western dating, and so I want to share their story and what I learned from it with you. For background, I'll start with telling you what arranged marriage is actually like.

Although some parts of India still do the traditional "bride and groom don't meet until the wedding", these tend to be remote and rural parts. Most arranged marriages today function a little more like a blind date, but with your parents and their network finding you a match rather than your friends. On the more traditional end, families may set up a "bride viewing", which today functions like a first meeting where the parents introduce each half of the couple, then leave them alone to get to know each other. They later tell their parents if they agree to the marriage or not. On the more liberal end, a couple may go on many dates before agreeing. In some cases, young people will date and fall in love, and the parents will meet after and decide to "arrange" the marriage if all parties agree to it. In the case of my parents, my dad's cousin (who he was very close with) met my mother and thought they would be a good match due to compatible philosophical interests and tastes in literature. My mother had, at that point, not dated at all, despite being in graduate school; it is normal for young people in India to feel marriage is not something they have to worry too much about because they trust their families will find someone good for them. The fact that my dad's cousin met my mother and immediately thought of my father

...
Do you like bullet points?
by Raemon — 2019-03-26T04:30:59.104Z

I think more naturally in bullet points, and I (sometimes) like reading posts that are written in bullet style. (This website is one of my favorites, and is written entirely in bullets).

(Disclaimer, although I wrote this post in bullet points because it was cute, I don't think it's the best exemplar of them. Or rather, it's an example of using bullet points to do rough thinking, rather than an example of using them to illustrate a complex argument)

I like bullet points because:

It's easier to skim, and build up a high level understanding of a post's structure. If you understand a concept you can skip it and move on, if you want to drill down and understand it better you can do so.

Relatedly, it exposes your cruxes more readily. You can pick out and refute points, in a way that can be harder with meandering prose.

It's easier to hash out early stage ideas. When I'm first thinking about something, my brain is jumping around and forming connections, developing a model at multiple levels of resolution. Bullet lists make this easier to keep track of.

I like this for other people's posts as well, since it feels more playful, like I can be part of their early generation process. I think LessWrong would be better if more people wrote more unpolished things to get early feedback on them, and bullet lists are a nice way to signal that something is still in development.

Prose often adds unnecessary cruft. In the transition from bullets-to-prose, posts can go 2x-3x as long (or, when I go to write a short bullet summary of something I wrote in prose, it turns out to be much shorter, and the prose mostly unnecessary)

I had assumed this was a common experience, and that it was in fact a weakness of humanity that we didn't have better, more comprehensive bullet-point tools.

But, alas, Typical Mind Fallacy. It turned out a couple people on the LessWrong team reacted very negatively to bullet points. Concerns include:

It's easy to think you've communicated more clearly than you have, because you didn't bother writing the connecting words between paragraphs.
They're harder to read straight through. If you include bold words, readers might not bother reading the non-bold words, and miss nuance.
"I like numbered arguments, since that makes it easier to respond to individual points. But unnumbered bullet lists are just h

...
IRL 4/8: Maximum Entropy IRL and Bayesian IRL
by RAISE — 2019-03-25T22:07:29.012Z
Link: https://app.grasple.com/#/level/1499

Every Monday for 8 weeks, we will be posting lessons about Inverse Reinforcement Learning. This is lesson 4.

Note that access to the lessons requires creating an account here.

This lesson comes with the following supplementary material:

The Principle of Maximum Entropy

Have a nice day!
If you've attended LW/SSC meetups, please take this survey!
by mingyuan — 2019-03-25T21:48:37.976Z

I've put together a survey to gather information on the state of meetups around the world, and in particular to figure out what kinds of actions it might be useful for people interested in global meetup coordination to take. It's branded as being for SlateStarCodex meetups, but please don't be put off by that if your group isn't affiliated with SSC - the branding is just an artifact of previous decisions, but I'm just as interested in getting data on LW and EA groups.

You can take the survey here.

Context: I've been organizing and thinking about meetups for a couple years now. I coordinated the SSC Meetups Everywhere 2018 and I received a grant from the Centre for Effective Altruism to coordinate SSC meetups.

--

Please let me know either in the survey or in the comments below if you have any feedback or questions! It's very unlikely that I'll make changes to the survey questions now since that would mess up the data, but this is my first time doing something like this and I will definitely take feedback into account for the future.

Data will not be released publicly because it would be too easy to identify individuals and I neglected to include a question about releasing people's answers, but I am planning to share aggregate statistics and lessons learned publicly. I will also probably reach out to individual meetup organizers if there's significant data on what people want to see from their groups.