trentmkelly/LessWrong-43k huggingface dataset:
How hard is it for altruists to discuss going against bad equilibria?
by abramdemski — 2019-06-22T03:42:24.416Z

Epistemic status: This post is flagrantly obscure, which makes it all the harder for me to revise it to reflect my current opinions. By the nature of the subject, it's difficult to give object-level examples. If you're considering reading this, I would suggest the belief signaling trilemma as a much more approachable post on a similar topic. Basically, take that idea, and extrapolate it to issues with coordination problems?

There are many situations where a system is "broken" in the sense that incentives push people toward bad behavior, but, not so much that an altruist has any business engaging in that bad behavior (at least, not if they are well-informed).

In other words, an altruist who understands the bad equilibrium well would disengage from the broken system, or engage while happily paying the cost of going against incentives.
Clearly, this is not always the case; I'm thinking about situations where it is the case.

Actually, I'm thinking about situations where it is the case supposing that we ignore certain costs, such as costs of going against peer pressure, costs of employing willpower to go against the default, etc. The question is then: is it realistically worth it, given all those additional costs, if we condition on it being it's worth it for an imaginary emotional-robot altruist?
Actually actually, the question I'm asking is probably not that one either, but I haven't figured out my real question yet.

I think maybe I'm mainly interested in the question of how hard it is for altruists to publicly discuss altruistic strategies (in the context of a bad equilibrium) without upsetting a bunch of people (who are currently coordinating on that equilibrium, and are therefore protective of it).

I'm writing this post to try to sort out some confused thoughts (hence the weird style). A lot of the context is discussion on this post.

But, I'm not going to discuss examples in my post. This seems like a case where giving examples is more likely to steer discussion to unproductive places than the reverse, at least if those examples are close to anyone's real situation/concern.

I'm using the term "altruist" in an absolute way here, which is a bit misleading.

I think it makes sense to talk about, and try to understand, what a perfect altruist can do to forward their own values. I'm not talking about a decision-theoretically perfect altruist. I'm talking about a basically

...
The Hacker Learns to Trust
by Ben Pace — 2019-06-22T00:27:55.298Z
Link: https://medium.com/@NPCollapse/the-hacker-learns-to-trust-62f3c1490f51

This is a linkpost for some interesting discussions of info security norms in AI. I threw the post below together in 2 hours, just to have a bunch of quotes and links for people, and to have the context in one place for a discussion here on LW (makes it easier for common knowledge of what the commenters have and haven't seen). I didn't want to assume people follow any news on LW, so for folks who've read a lot about GPT-2 much of the post is skimmable.

Background on GPT-2

In February, OpenAI wrote a blogpost announcing GPT 2:

We’ve trained a large-scale unsupervised language model which generates coherent paragraphs of text, achieves state-of-the-art performance on many language modeling benchmarks, and performs rudimentary reading comprehension, machine translation, question answering, and summarization—all without task-specific training.

This has been a very important release, not least due to it allowing fans to try (and fail) to write better endings to Game of Thrones. Gwern used GPT-2 to write poetry and anime. There have been many Medium posts on GPT-2, some very popular, and at least one Medium post on GPT-2 written by GPT-2. There is a subreddit where all users are copies of GPT-2, and they imitate other subreddits. It got too meta when the subreddit imitated another subreddit about people play-acting robots-pretending-to-be-humans. Stephen Woods has lots of examples including food recipes.

Here in our rationality community, we created user GPT-2 trained on the entire corpus of LessWrong comments and posts and released it onto the comment section on April 1st (a user who we warned and then banned). And Nostalgebraist created a tumblr trained on the entire writings of Eliezer Yudkowsky (sequences+HPMOR), where Nostalgebraist picked their favourites to include on the Tumblr.

There was also very interesting analysis on LessWrong and throughout the community. The post that made me think most on this subject is Sarah Constantin's Human's Who Are Not Concentrating Are Not General Intelligences. Also see SlateStarCodex's Do Neural Nets Dream of Electric Hobbits? and GPT-2 As Step Toward General Intelligence, plus my teammate jimrandomh's Two Small Experiments on GPT-2.

However, these were all using a nerfed version of GPT-2, which only had 175 million parameters, rather than the fully trained model with 1.5 billion parameters. (...
"The Bitter Lesson", an article about compute vs human knowledge in AI
by the gears to ascension — 2019-06-21T17:24:50.825Z
Link: http://www.incompleteideas.net/IncIdeas/BitterLesson.html

The Bitter Lesson
Rich Sutton

The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin. The ultimate reason for this is Moore's law, or rather its generalization of continued exponentially falling cost per unit of computation. Most AI research has been conducted as if the computation available to the agent were constant (in which case leveraging human knowledge would be one of the only ways to improve performance) but, over a slightly longer time than a typical research project, massively more computation inevitably becomes available. Seeking an improvement that makes a difference in the shorter term, researchers seek to leverage their human knowledge of the domain, but the only thing that matters in the long run is the leveraging of computation. These two need not run counter to each other, but in practice they tend to. Time spent on one is time not spent on the other. There are psychological commitments to investment in one approach or the other. And the human-knowledge approach tends to complicate methods in ways that make them less suited to taking advantage of general methods leveraging computation. There were many examples of AI researchers' belated learning of this bitter lesson, and it is instructive to review some of the most prominent.

In computer chess, the methods that defeated the world champion, Kasparov, in 1997, were based on massive, deep search. At the time, this was looked upon with dismay by the majority of computer-chess researchers who had pursued methods that leveraged human understanding of the special structure of chess. When a simpler, search-based approach with special hardware and software proved vastly more effective, these human-knowledge-based chess researchers were not good losers. They said that brute force" search may have won this time, but it was not a general strategy, and anyway it was not how people played chess. These researchers wanted methods based on human input to win and were disappointed when they did not.

A similar pattern of research progress was seen in computer Go, only delayed by a further 20 years. Enormous initial efforts went into avoiding search by taking advantage of human knowledge, or of the special features of the game, but all those efforts proved irrelevant, or worse, once search was app

...
Decisions are hard, words feel easier
by Hazard — 2019-06-21T16:17:22.366Z

Categorization often serves the purpose of making decisions. Recall, making decisions is hard. Sometimes it's easier (but not necessarily more effective) to think about if a word applies, rather than to think directly about how a decision should be made. This happens on the in individuals and in groups.

Groups

Explicit note: When a word appears in quotation marks, I'm referring to the English word itself. When it appears unquoted, I'm referring to the concept it invokes.

Read this carefully. This post makes claims about "racism", not many about racism. If you disagree, please use your best inference powers to engage with what I mean, lest I become a "racist".

"Racism". Not racism, but "racism".

What do I know about someone who is racist? Depends on what I meant. What do I know about someone who is "racist"? I know that:

If I associate with them I'm in danger of being attacked.

Other people consider them unsavory.

The move to notice is that regardless of whatever racism actually is, there's a decently stable way that the word "racism" works in America. This is the lawyer. If the word sticks, game moves are allowed to be made. I'm allowed to dox someone on the internet for being "racist". I'm not allowed to dox someone on the internet who isn't "racist". But am I allowed to dox someone o the internet who is racist? Hard to say, unless I know they're "racist". Until that battle has been fought, I don't know what's going to happen.

My argument is not that you shouldn't use the word racism like this, but to make clear the actual cognition and social inferences that are happening. Defining racism is not a "just for fun" intellectual exercise. Through inter-subjective magic, the word "racism" has power. Maybe it should, maybe it shouldn't. Either way, it does.

Oops, we've got a situation on our hands that incentives being a lawyer. All parties know that if there is loud consensus on X being "racist", a certain outcome will happen. All parties want certain outcomes. Arguing about if the word "racist" applies becomes such a direct path to getting what one wants, why go a different route?

Being "racist" means that a lot of people agreed to use the word "racist" to talk about you. Being racist means [something].

Now, the "if... then..." decision rules are no longer applied to the neural categories. They are applied to things that the social fabric allows to be called a WORD

...
Splitting Concepts
by Hazard — 2019-06-21T16:03:11.177Z

For anyone who's gone through some form of primary education, I'd expect numbers to be a tight neural category. Numbers feel like things, they are pretty much the same, and there's little ambiguity over if something is a number or not.

So it might come to one's surprise when they learn about the ordinal number omega which has the interesting property:
ω<ω+1|ω|=|ω+1|
Where "<" means "less than" and "|x|" means "the size of the number x"

You're telling me there's a number that's less than another number, but still the same size as another number? Bullshit, quit pulling my leg.

A mathematician will quickly tell you that I got the meaning the above symbols wrong. "<" talks about order and "|x|" talks about cardinality. Ordinality refers to somethings position in a list relative to other elements. You can come before another element or after another element.

Cardinality refers to being "bigger" or "smaller".

Here's another fun fact:
ω+1≠1+ω
Okay, now you're just shitting me. This might as well not be a number!

This annoy someone for a few reasons:

In the numbers you're used to, "comes before" and "is smaller than" always meant the same thing.

You don't even remember the phrase "communative" and the second piece of math just looks fake.

In general, though these squiggly things look like the numbers you're used to, they sure don't act like them.

You are being told to split a concept. This is always a bit difficult, and a source of mental resistance. You have to literally rewire your brain to prevent certain inferences being made. Rewiring = work = experience of effort.

If numbers never felt like yours to begin with, this will be easier. If you don't care about numbers that much you may resent your teacher for "adding" another cognitive stumbling block, but you will proceed.

If you feel like you "know what a number is", this will be harder. Even if you learn the math of ordinals, you will wail and gnash your teeth every time someone refers to them as numbers. This is the mistake of the student. Fine, naturals and ordinals are different, but we can't use the word number for both of them, because what if I was asked this question on a test and wasn't able to give the right answer?!?

Mathematicians redefine words all the time. In this way, they are like poets. They can get away with this because this is what they do. In school you will be told that if you intuition

...
Modeling AGI Safety Frameworks with Causal Influence Diagrams
by Ramana Kumar — 2019-06-21T12:50:08.233Z
Link: https://arxiv.org/abs/1906.08663

We have written a paper that represents various frameworks for designing safe AGI (e.g., RL with reward modeling, CIRL, debate, etc.) as Causal Influence Diagrams (CIDs), to help us compare frameworks and better understand the corresponding agent incentives.

We would love to get comments, especially on

Are the depicted frameworks represented accurately?
Is the CID representation helpful?
Frameworks we did not include that would be useful to model this way?

The paper's abstract:

Proposals for safe AGI systems are typically made at the level of frameworks, specifying how the components of the proposed system should be trained and interact with each other. In this paper, we model and compare the most promising AGI safety frameworks using causal influence diagrams. The diagrams show the optimization objective and causal assumptions of the framework. The unified representation permits easy comparison of frameworks and their assumptions. We hope that the diagrams will serve as an accessible and visual introduction to the main AGI safety frameworks.
Is your uncertainty resolvable?
by Raemon — 2019-06-21T07:32:00.819Z

I was chatting with Andrew Critch about the idea of Reacts on LessWrong.

Specifically, the part where I thought there are particular epistemic states that don’t have words yet, but should. And that a function of LessWrong might be to make various possible epistemic states more salient as options. You might have reacts for “approve/disapprove” and “agree/disagree”... but you might also want reactions that let you quickly and effortless express “this isn’t exactly false or bad but it’s subtly making this discussion worse.”

Fictionalized, Paraphrased Critch said “hmm, this reminds me of some particular epistemic states I recently noticed that don’t have names.”

“Go on”, said I.

“So, you know the feeling of being uncertain? And how it feels different to be 60% sure of something, vs 90%?”

“Sure.”

“Okay. So here’s two other states you might be in:

75% sure that you’ll eventually be 99% sure,
80% sure that you’ll eventually be 90% sure.

He let me process those numbers for a moment.

...

Then he continued: "Okay, now imagine you’re thinking about a particular AI system you’re designing, which might or might not be alignable.

“If you’re feeling 75% sure that you’ll eventually be 99% sure that that AI is safe, this means you think that eventually you’ll have a clear understanding of the AI, such that you feel confident turning it on without destroying humanity. Moreover you expect to be able to convince other people that it’s safe to turn it on without destroying humanity.

“Whereas if you’re 80% sure that eventually you’ll be 90% sure that it’ll be safe, even in the future state where you’re better informed and more optimistic, you might still not actually be confident enough to turn it on. And even if for some reason you are, other people might disagree about whether you should turn it on.

“I’ve noticed people tracking how certain they are of something, without paying attention to whether their uncertainty is possible to resolve. And this has important ramifications for what kind of plans they can make. Some plans require near-certainty. Especially many plans that require group coordination.

“Makes sense”, said I. "Can I write this up as a blogpost?"

I’m not quite sure about the best name here, but this seems like a useful concept to have a handle for. Something like “unresolvable uncertainty?”
Should rationality be a movement?
by Chris_Leong — 2019-06-20T23:09:10.555Z

This post is a quick write-up of a discussion that I recently had with two members of the rationality community. For reasons of simplicity I'll present them as holding a single viewpoint that is a merger of both their arguments. All parties seemed to be in agreement about the long-term future being an overwhelming consideration, so apologies in advance to anyone with a different opinion.

In a recent discussion, I noted that the rationality community didn't have an organisation like CEA engaging in movement building and suggested this might at least partially why EA seemed to be much more successful than the rationality community. While the rationality community has founded the MIRI and CFAR, I pointed out that there were now so many EA-aligned organisations it's impossible to keep track. EA runs conferences where hundreds of people attend, with more on the waitlist, while LW doesn't even have a conference in it's hometown. EA has groups at the most prominent universities, while LW has almost none. Further, EA now has it's own university department at Oxford and the support of OpenPhil, a multi-billion dollar organisation. Admittedly, Scott Alexander grew out of the rationality community, but EA has 80,000 hours. I also noted that EA had created a large number of people who wanted to become AI safety researchers; indeed at some EA conferences it felt like half the people there were interested in pursuing that path.

Based on this comparison, EA seems to have been far more successful. However, the other two suggested that appearances could be misleading and that it therefore wasn't so obvious that rationality should be a movement at all. In particular, they argued that most of the progress made so far in terms of AI safety didn't come from anything "mass-movement-y".

For example, they claimed:

Slatestarcodex has been given enthusiastic praise by many leading intellectuals who may go on to influence how others think. This is the work of just one man who has intentionally tried to limit the growth of the community around it
Eliezer Yudkowsky was more influential than EA on Nick Bostrom's Superintelligence. This book seems to have played a large role in convincing more academic types to take this viewpoint more seriously. Neither Yudkowsky's work on Less Wrong nor Superintelligence are designed for a casual au

...
Let Values Drift
by Gordon Seidoh Worley — 2019-06-20T20:45:36.618Z

I occasionally run across lines of reasoning that depend on or favor the position that value drift should be avoided.

I find odd the idea of value drift, let alone the idea that value drift is bad. My intuition is that value drift is good if anything since it represents an update of one's values based on new evidence and greater time to compute reflective equilibrium. But rather than arguing intuition, let's explore value drift a bit before we come to any stronger conclusions.

(Fair warning, this is going to get into some deep philosophical territory, be pretty unapologetic about it, and assume you are reading carefully enough to notice what I say and not what you think I said. I'm still working some of these ideas out myself, so I don't have the fluency to provide a more accessible explanation right now. I also take some pretty big inferential jumps at times that you may not be on board with as of yet, so the later parts might feel like unjustified reasoning. I don't think that's the case, but you'll have to poke at me to help me figure out how to fill in those gaps.

In spite of all those apologies, there are some key insights here, and I'm unlikely to get clearer unless I am first more opaque, so please bear with me if you please, especially if you are interested in value as it relates to AI alignment.)

Whence drifting values?

The metaphor of drifting values is that your values are initially one place and then gradually relocate to another, like flotsam. The waves of fortune, chance, and intention combine to determine where they end up on the seas of change. In this metaphor, values are discrete, identifiable things. Linguistically, they are nouns.

When we talk of values as nouns, we are talking about the values that people have, express, find, embrace, and so on. For example, a person might say that altruism is one of their values. But what would it mean to "have" altruism as a value or for it to be one of one's values? What is the thing possessed or of one in this case? Can you grab altruism and hold onto it, or find it in the mind cleanly separated from other thoughts? As best I can tell, no, unless contrary to evidence and parsimony something like Platonic idealism proves consistent with reality, so it seems a type error to say you possess altruism or any other value since values are not things but habit...
A case for strategy research: 
what it is and why we need more of it
by Siebe — 2019-06-20T20:22:14.478Z

Authors: Siebe Rozendal, Justin Shovelain, David Kristoffersson

Crossposted to the Effective Altruism Forum

Overview

To achieve any ambitious goal, some strategic analysis is necessary. Effective altruism has ambitious goals and focuses heavily on doing research. To understand how to best allocate our time and resources, we need to clarify what our options in research are. In this article, we describe strategy research and relate it to values research, tactics research, informing research, and improvement research. We then apply the lens of strategy research to existential risk reduction, a major cause area of effective altruism. We propose a model in which the marginal value of a research type depends strongly on the maturity of the research field. Finally, we argue that strategy research should currently be given higher priority than other research in existential risk reduction because of the significant amount of strategic uncertainty, and we provide specific recommendations for different actors.

Introduction

Effective altruism is regularly framed as “figuring out how to do the most good, and then doing it.” However, figuring out how to do the most good is not easy. Different groups reach different conclusions. So how do we figure out how to do the most good?

Quite obviously, the first step is to figure out our values. We need to know what we roughly mean by ‘the most good.’ However, once our moral uncertainty is significantly diminished, what is the next step in figuring out how to do the most good? We believe the next step should be strategy research: high-level research on how to best achieve a high-level goal. A brief case was made for strategic analysis by Nick Bostrom in Superintelligence (p. 317):

"Against a backdrop of perplexity and uncertainty, [strategic] analysis stands out as being of particularly high expected value. Illumination of our strategic situation would help us target subsequent interventions more effectively. Strategic analysis is especially needful when we are radically uncertain not just about some detail of some peripheral matter but about the cardinal qualities of the central things. For many key parameters, we are radically uncertain even about their sign…”

This was written in the context of existential risk from AI. We believe it applies to existential risks in general, and that strategy research should be a core part of othe...
LW Dev question: FB-style tagging?
by Dr_Manhattan — 2019-06-20T19:19:45.807Z

As LW is an intellectual **community** where people get to know each other and know "who knows what, who's working on what" it seems like something like tagging people FB-style in specific posts/comments is a very low-hanging fruit. (Of course you should be able to silence people from tagging you or sort taggers by some score). Is this in the pipeline, or has been ruled out for some reason?