trentmkelly/LessWrong-43k huggingface dataset:
Combat vs Nurture: Cultural Genesis
by Ruby — 2018-11-12T02:11:42.921Z

In my post

Conversational Cultures: Combat vs Nurture

, I described two different sets of norms and assumptions norms used in discussion. In this follow-up post, I add some important clarifications, state the defining differences, and begin to explore the conditions which might give rise to each culture.

What these “cultures” are and are not

Though I have written as though there are these two distinct neat “cultures”, there are, of course, several giant fuzzy overlapping clusters of behaviors and correlated traits among people in this space of combat/nurture/etc. The specific clusters of behavior which I want to discuss are those related to the discussion of ideas, communication of information, and the ostensible goal of reaching agreement either about matters of fact or action to be taken.

Adjacent to these clusters is a host of broader cultural behaviors. For example, New Yorkers have a reputation for being more

candid/impatient/blunt/arrogant/pushy

than most. While also sociologically interesting, this post and my last post aren’t about the general spectrum of blunt/direct vs. polite/friendly, etc.

Lastly, the names I’ve used for the cultures are pretty fuzzy. They’re more successful at being easy to say and evocative than being definitely the best English words to point at the thing. “Adversarial”, “Direct”, “Cooperative”, “Collaborative”, and “Polite” is just a starting list the viable alternatives for names of the cultures.

Evaluations of the cultures

To be more prescriptive than I was in my last post, I want to be clear that I think there exist instantiations of both Combat and Nurture culture which are “relatively healthy”, i.e. their practitioners are benevolent, mostly not harmed by the culture, and they succeed at communication. While they’re both far from optimal as usually practiced, I strongly disagree with those who see one culture as deleterious and dysfunctional and the other as the obviously healthy and right one. I think that's true despite it being easy to find particular instances where each culture goes very wrong.

Perhaps it is predictable and cliche to have this opinion, but whatever the ideal communication culture is, it is going to involve modeling (and combining) behaviors from both each of the cultures. It’s probable, in fact, that no actual real-world functioning culture consists solely of people embodying acts from only one or ...
Rationality Is Not Systematized Winning
by namespace — 2018-11-11T22:05:19.153Z
Link: http://www.thelastrationalist.com/rationality-is-not-systematized-winning.html

"Rationality is systematized winning" is a slogan that was adopted to patch a bug in human cognition. Namely our endless capacity to delude ourselves about how we did in an attempt to save face. The concept seems to have been absorbed, but I'm skeptical it's translated into more effective action. Certainly it produced many essays on why winning isn't happening. But the fact that we've been publishing essentially the same essay for a decade now implies something fairly fundamental is wrong. This slogan was chosen because it patches the bug, but I fear at the cost of neutering our ability to focus.
“She Wanted It”
by sarahconstantin — 2018-11-11T22:00:01.645Z

Epistemic Status: Things I Will Regret Writing

One of the most persistent arguments, often left implicit, that antifeminists have on their side is “Women want to be raped and abused.”

On one hand, this is an obvious logical contradiction if taken literally. Rape is unwanted sex — how can you want something unwanted?

On the other hand, I can see why people might think this.

Don Giovanni, one of the classic archetypes of a man who’s attractive to women, is also a rapist. It’s not ambiguous — in the opera, he breaks into a woman’s house to rape her while she fights him off. No distinction is made by the librettist between his “seductive” charms and his violent attack; he goes to Hell for being a “libertine”, a sin that includes both. We have a lot of art, often but not always created by men, that portrays male cruelty and male attractiveness as one and the same.

Men are more violent than women. This is a human universal, and true in many of our mammalian relatives as well. And success in violent conflict is, of course, an advantage for inclusive fitness, so there’s an evolutionary rationale for an attraction to men who are strong and good at winning fights.

From there, anti-feminists often jump to believing that women are more attracted to men who are violent to them. This doesn’t directly follow, of course, but then you point to certain cultural trends — the high incidence of rape fantasies, the fact that many violent criminals have no difficulty finding female partners, the phenomenon of women who have been serially abused by multiple partners, the popularity of Fifty Shades of Gray — and you might start to wonder whether at least some women might have a thing for men who use force on them.

I think you have to talk about this thing frankly before you can put it to rest.

So let’s talk about Fifty Shades of Grey.

It’s a popular book, selling over 125 million copies worldwide. It was also written by a self-published author and gained popularity through word of mouth; almost all critics reviewed it poorly; so its popularity is an unusually strong signal that people genuinely like it. Nobody is buying this book to impress people.

And it’s a description of a woman in an abusive relationship with a rapist. I recommend Pervocracy’s very funny liveblog of the book for reference.

The question Pervocracy asks is — why are people into this? Sure, there’s sex and BDSM with a...
Future directions for ambitious value learning
by Rohin Shah — 2018-11-11T15:53:52.888Z

To recap the sequence so far:

Ambitious value learning aims to infer a utility function that is safe to maximize, by looking at human behavior.
However, since you only observe human behavior, you must be able to infer and account for the mistakes that humans make in order to exceed human performance. (If we don’t exceed human performance, it’s likely that we’ll use unsafe techniques that do exceed human performance, due to economic incentives.)
You might hope to infer both the mistake model (aka systematic human biases) and the utility function, and then throw away the mistake model and optimize the utility function. This cannot be done without additional assumptions.
One potential assumption you could use would be to codify a specific mistake model. However, humans are sufficiently complicated that any such model would be wrong, leading to model misspecification. Model misspecification causes many problems in general, and is particularly thorny for value learning.

Despite these arguments, we could still hope to infer a broad utility function that is safe to optimize, either by sidestepping the formalism used so far, or by introducing additional assumptions. Often, it is clear that these methods would not find the true human utility function (assuming that such a thing exists), but they are worth pursuing anyway because they could find a utility function that is good enough.

This post provides pointers to approaches that are currently being pursued. Since these are active areas of research, I don’t want to comment on how feasible they may or may not be -- it’s hard to accurately assess the importance and quality of an idea that is being developed just from what is currently written down about that idea.

Assumptions about the mistake model. We could narrow down on the mistake model by making assumptions about it, that could let us avoid the impossibility result. This decision means that we’re accepting the risk of misspecification -- but perhaps as long as the mistake model is not too misspecified, the outcome will still be good.

Learning the Preferences of Ignorant, Inconsistent Agents shows how to infer utility functions when you have an exact mistake model, such as “the human is a hyperbolic time discounter”. (Learning the Preferences of Bounded Agents and the online book Modeling Agents with Probabilistic Programs cover similar ground.)

Inferring Reward Functions ...
Reconciling Left and Right, from the Bottom-Up
by jesseduffield — 2018-11-11T08:36:05.554Z

This is a repost of my blog post from my medium account, . I got the idea to write it based on this comment I left on Vaniver's post 'Public Positions and Private Guts'. This is the first thing I've posted on LessWrong so hopefully it's fitting the guidelines.

I think right now a lot of things are converging in terms of how we conceptualise the processes of the brain. Conscious vs Subconscious, Science vs Art, Abstract vs Concrete, Left vs Right, what if these dichotomies were all about the same thing? I think the real dichotomy happening with all these things is actually a dichotomy between top-down and bottom-up processing.

Top-down processing is how you get from abstract concepts like goals and aspirations to concrete-world implementations, like deciding you want to bake a cake, and then reading a recipe and performing each step of the recipe until you eventually have a cake. Bottom-down processing is how you see a messy complex data set like an image of billions of tiny yellow rocks meeting an oscillating body of blue liquid and instinctively think ‘beach’. Top-down processing is generative: you start from something simple and end up with something complex. Bottom-up is reductive, you start with something complex and you reduce it down to something simple.

To give a word to a pattern you’ve identified is to grant it ascension into the realm of top-down processing, where you can now apply things like logic and deductive reasoning to achieve goals with that abstracted pattern. For example, you might be playing several 1v1 games of age of empires against the same opponent and realise ‘this guy keeps attacking me early in the game’ and then you might come up with a word for that, for example you could say that you’re being ‘rushed’ and now you’ve brought a fairly complicated and messy concept into the clean, manageable world of abstract thought. Now you can use predicate logic, whose utility extends to any set of abstract concepts you want to make statements about, by saying ‘if I get rushed, I need a strategy to counter that’. So you’ve now set a goal, and you can use top-down reasoning to bring that goal closer to the concrete world by thinking ‘well I can make sure I put some watchtowers up early so that I have some defensive units without too much cost to my economy’. Now you can go back up the abstraction ladder and call that defe...
Competitive Markets as Distributed Backprop
by johnswentworth — 2018-11-10T16:47:37.622Z

Requisite background: high school level programming and calculus. Explanation of backprop is included, skim it if you know it. This was originally written as the first half of a post on organizational scaling, but can be read standalone.

Backpropagation

If you’ve taken a calculus class, you’ve probably differentiated functions like g(x)=(x2+1)e−x22. But if you want to do math on a computer (for e.g. machine learning) then you’ll need to differentiate functions like

function f(x): # Just wait, the variable names get worse…
a = x^(1/2) # Step 1
b = 1–1/(a+1) # Step 2
c = ln(b) # Step 3
return c

This is a toy example, but imagine how gnarly this could get if f(x) contains loops or recursion! How can we differentiate f(x), without having to write it all out in one line? More specifically, given the code to compute f(x), how can we write code to compute dfdx? The answer is backpropagation. We step through the function, differentiating it line-by-line, in reverse. At each line, we compute the derivative with respect to one of the internal variables. As we work back, we differentiate with respect to earlier and earlier variables until we reach x itself.

The key to backprop is the chain rule. Look the bottom left arrow: dfdx=dfdadadx. That’s just the chain rule! A similar formula applies to each arrow in the bottom row.

Let’s work through the example. Keep an eye on the diagram above to see where we are as we go. First, we differentiate the last line:

return c: df_dc = 1

Then, we work backward:

Step 3: c = ln(b) -> df_db = df_dc * dc_db = df_dc ∗ (1/b)

Notice what happened here: we used the chain rule, and one of the two pieces to come out was dfdc — the result of the previous step. The other piece is the derivative of this particular line. Continuing:

Step 2: b = 1–1/(a+1) -> df_da = df_db * db_da = df_db ∗ (1/(a+1)^2)
Step 1: a = x^(1/2) -> df_dx = df_da * da_dx = df_da ∗ (1/2*x^(−1/2))

Each step takes the result of the previous step and multiplies it by the derivative of the current line. Now, we can assemble it all together into one function:

function df_dx(x):
a = x^(1/2) # Step 1
b = 1–1/(a+1) # Step 2
c = ln(b)

...
Productivity: Instrumental Rationality
by frdk666 — 2018-11-10T14:58:36.336Z

Meta: I apologise if this is not of interest to most people on LW. I am posting this here anyway since I estimated there was at least a 5% chance that I would find what I'm looking for here, with minimal investment of my time.

Is anyone here tracking productivity every day? Do you have a fixed schedule? Are you debugging the causes for deviations from schedule and fixing them? What is your goal and why do you think the path you've chosen is the best way to achieve that?

I started tracking productivity last week. My last week's average was 6.20 hours per day (including weekends). I've already improved from around 3 hours a day (I'm employed as a software developer, I write backend APIs in Go for a fairly large company).

The existential dread of writing one technically unchallenging API after the next has finally gotten to me, and I want to drastically change my situation as quickly as possible. I started a side project which is making slow progress -- I don't feel comfortable revealing the specifics online, but I'm looking for someone who has a fixed schedule and tracks productivity every day (it's nice to have social feedback). So far, I've been spamming one of my friends for social feedback -- but it's really not working since he's not as dedicated.

My goal productivity wise is to hit 12 hours / day consistently (or an 80 hour week). I expect to get here faster if someone else is also doing the same thing. To be more specific, the workflow I have in mind is send a message after every 2 hours of productive work, talk about how much progress you've made, and stress test the path you've chosen with constructive negative feedback.

I realise this is not exactly balanced etc., but I've already decided I want to do this -- so please assume that the productivity goal is a fixed axiom if you decide to comment on this. Thanks!
Preface to the sequence on iterated amplification
by paulfchristiano — 2018-11-10T13:24:13.200Z

This sequence describes iterated amplification, a possible strategy for building an AI that is actually trying to do what we want out of ML systems trained by gradient descent.

Iterated amplification is not intended to be a silver bullet that resolves all of the possible problems with AI; it’s an approach to the particular alignment problem posed by scaled-up versions of modern ML systems.

Iterated amplification is based on a few key hopes

If you have an overseer who is smarter than the agent you are trying to train, you can safely use that overseer’s judgment as an objective.
We can train an RL system using very sparse feedback, so it’s OK if that overseer is very computationally expensive.
A team of aligned agents may be smarter than any individual agent, while remaining aligned.

If all of these hopes panned out, then at every point in training “a team of the smartest agents we’ve been able to train so far” would be a suitable overseer for training a slightly smarter aligned successor. This could let us train very intelligent agents while preserving alignment (starting the induction from an aligned human).

Iterated amplification is still in an preliminary state and is best understood as a research program rather than a worked out solution. Nevertheless, I think it is the most concrete existing framework for aligning powerful ML with human interests.

Purpose and audience

The purpose of this sequence is to communicate the basic intuitions motivating iterated amplification, to define iterated amplification, and to present some of the important open questions.

I expect this sequence to be most useful for readers who would like to have a somewhat detailed understanding of iterated amplification, and are looking for something more structured than ai-alignment.com to help orient themselves.

The sequence is intended to provide enough background to follow most public discussion about iterated amplification, and to be useful for building intuition and informing research about AI alignment even if you never think about amplification again.

The sequence will be easier to understand if you have a working understanding of ML, statistics, and online learning, and if you are familiar with other work on AI alignment. But it would be reasonable to just dive in and just skip over any detailed discussion that seems to depend on missing prerequisites.

Outline and reading recommendations

Th

...
Specification gaming examples in AI
by Samuel Rødal — 2018-11-10T12:00:29.369Z
Link: https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vRPiprOaC3HsCf5Tuum8bRfzYUiKLRqJmbOoC-32JorNdfyTiRRsR7Ea5eWtvsWzuxo8bjOxCG84dAg/pubhtml

Interesting list of examples where AI programs gamed the specification, solving the problem in rather creative (or dumb) ways not intended by the programmers.
Real-time hiring with prediction markets
by ryan_b — 2018-11-09T22:10:18.576Z

Epistemic status: speculation

I have a couple of assumptions about hiring:

It is difficult and often unpleasant.
No one is very good at it.
It has a lot of hidden costs.

I have a further assumption about the need for labor:

The people doing the work now have the best information about the amount and type of work needed.

Most of the time I have seen a new position open up, it worked approximately like this: management notices a shortfall -> they make a case for a new position -> there will be negotiation with senior management/HR/finance about the needs and available resources -> greenlight to post a job opening and collect applications. This seems like costs begin to pile up until management takes notice, and then continue during the hiring process. The goal is usually to get a pool of qualified applicants, and then make the lowest offer that will get accepted. Controlling explicit costs like compensation seems to be the dominant concern; there is little to none for lost productivity.

I think if a prediction market were implemented where current employees bet on the value of applicants (their future colleagues), we could get better hiring decisions. If we combine this with continuous hiring, which is to say always accepting applications, we can cut out the entire above process.

As new applications come in, the market will price them. A high price is worth a look, because it means the team wants to add them. Price inflation overall implies the team is in need - though I am not clear on where additional funds would come from.

It seems like this leads to a condition where a company could hire productive people as soon as it needs them, without even having to recognize that need. This sounds ideal to me.
Current AI Safety Roles for Software Engineers
by ozziegooen — 2018-11-09T20:57:16.159Z

[Note: Please make sure to see the comments for other, newer information]

I've had several conversations over the last few months with engineers who were trying to enter the field of AI safety. It became evident that I was giving pretty much the same advice to all of them, so I finally decided to write it up.

Some more context; late last year it became evident to me that funding was becoming much less of an issue around AI safety and engineering expertise was becoming more of an issue. I decided to leave my job to work in the area. I spent some time consulting for Ought, then eventually came to a point where it seemed more useful to do some self-studying for a while. During that period I spoke to several people in the Bay Area about engineering needs at AI safety organizations.

The hiring situation still seems a bit confusing to me. There are a lot of EA engineers who seem to want to do direct EA work, but are not sure what jobs they could get. Most AI safety organizations seem to really desire to find more good employees (and the AI-oriented ones, engineers), but are still fairly selective in their choices. I think that these organizations have typically been able to be selective, would prefer to do so when possible, and also have special demands that come from being small, new, and theoretical / EA.

If you are an engineer desiring to work in an EA organization soon or in the future, I suggest either getting really good at a few skills particularly useful to EA organizations (reinforcement learning, functional programming, ML), getting really good at startup engineering skills, or getting good at non-engineering skills desired by EA organizations. From what I've seen, spending marginal years on "generic medium-large company backend skills" is often not that useful for future EA positions at this point or expected to be in the future.

The following list represents the main organizations I've considered for work around AI safety, starting as an engineer without particular ML experience. If you are interested in all eng. positions in EA, I recommend 80k's job list. Also, 80,000 hours recently released an interview with two ea-aligned ML engineers, which I recommend if you are interested in more detail.

OpenAI Safety Team

I think the OpenAI safety initiatives may be some of the most visible AI-safety work at the moment. I believe the team has around 4-7 researchers

...