trentmkelly/LessWrong-43k huggingface dataset:
Forum participation as a research strategy
by Wei Dai — 2019-07-30T18:09:48.524Z

Previously: Online discussion is better than pre-publication peer review, Disincentives for participating on LW/AF

Recently I've noticed a cognitive dissonance in myself, where I can see that my best ideas have come from participating on various mailing lists and forums (such as cypherpunks, extropians, SL4, everything-list, LessWrong and AI Alignment Forum), and I've received a certain amount of recognition as a result, but when someone asks me what I actually do as an "independent researcher", I'm embarrassed to say that I mostly comment on other people's posts, participate in online discussions, and occasionally a new idea pops into my head and I write it down as a blog/forum post of my own. I guess that's because I imagine it doesn't fit most people's image of what a researcher's work consists of.

Once I noticed this, the tension is easy to resolve - in this post I'm going to proclaim/endorse forum participation (aka commenting) as a productive research strategy that I've managed to stumble upon, and recommend it to others (at least to try). Note that this is different from saying that forum/blog posts are a good way for a research community to communicate. It's about individually doing better as researchers.

Benefits of Forum Participation (FP)

FP takes little effort / will power

In other words it feels more like play than work, which means I rarely have issues with not wanting to do something that I think is important to do (i.e., akrasia), the only exception being that writing posts seems to take more effort so occasionally I spend my time writing comments when I perhaps should write posts instead. (This is the part of this post that I think may be least likely to generalize to other people. It could be that I'm an extreme outlier in finding FP so low-effort. However it might also be the case that it becomes low effort for most people to write comments once they've had enough practice in it.)

FP is a good way to notice missing background knowledge and provides incentives to learn missing knowledge

If you read a post with an intention to question or comment on it, it's pretty easy to notice that it assumes some background knowledge that you lack. The desire to not ask a "stupid" question or make a "stupid" comment provides powerful incentive to learn the miss knowledge.

FP is a good way to stay up to date on everyone else's latest research

It's often a g

...
Walkthrough: The Transformer Architecture [Part 1/2]
by Matthew Barnett — 2019-07-30T13:54:14.406Z

This is the first post in a new sequence in which I walk through papers, concepts, and blog posts related to machine learning and AI alignment.

In the process of increasing my own understanding of these topics, I have decided to implement a well known piece of advice: the best way to learn is to teach. This sequence is therefore a culmination of what I have learned or reviewed very recently, put into a learning format.

My thoughts on various topics are not polished. I predict there will be mistakes, omissions, and general misunderstandings. I also believe that there are better places to learn the concepts which I walk through. I am not trying to compete with anyone for the best explanations. The audience I have in mind is a version of myself from a few weeks or days ago, and therefore some things which I take for granted may not be common knowledge to many readers. That said, I think this sequence may be useful for anyone who wants a deeper look at the topics which I present.

If you find something wrong, leave a comment. Just try to be respectful.

If you've been following machine learning or natural language processing recently, you will likely already know that the Transformer is currently all the rage. Systems which are based on the Transformer have struck new records in natural language processing benchmarks. The GLUE benchmark is currently filled by models which, as far as I can tell, are all based on the Transformer.

With the Transformer, we now have neural networks which can write coherent stories about unicorns in South America, and an essay about why recycling is bad for the world.

The paper describing the Transformer, Attention Is All You Need, is now the top paper on Arxiv Sanity Preserver, surpassing the popularity of GANs and residual nets. And according to Google scholar, the paper has attracted 2588 citations (and counting).

So what makes the Transformer so powerful, and why is it such a break from previous approaches? Is it all just hype? Clearly not. But it is worth looking into the history behind the architecture first, in order to see where these ideas first came from. In this post, I give a rough sketch for what the transformer architecture looks like. In the following post, I will provide a detailed description of each step in the forward pass of the architecture. First, however, we have to know what we are looking at.

The central heart ...
Conversation on forecasting with Vaniver and Ozzie Gooen
by Bird Concept — 2019-07-30T11:16:58.633Z

[Cross-posted to the EA Forum]

This is a transcript of a conversation on forecasting between

Vaniver

and

Ozzie Gooen

, with an anonymous facilitator (inspired by the

double crux technique

). The transcript was transcribed using

a professional service

.

We decided to record, transcribe, edit and post it as:

Despite an increase in

interest

and

funding

for forecasting work in recent years, there seems to be a disconnect between the mental models of the people working on it and the people who aren’t. I want to move the community’s frontier of insight closer to that of the forecasting subcommunity
I think this is true for many more topics than forecasting. It’s incredibly difficult to be exposed to the frontier of insight unless you happen to be in the right conversations, for no better reason than that people are busy, preparing transcripts takes time and effort, and there are no standards and unclear expected rewards for doing so. This is an inefficiency in the economic sense. So it seems good to experiment with ways of alleviating it
This was a high-effort activity where two people dedicated several hours to collaborative, truth-seeking dialogue. Such conversations usually look quite different from comment sections (even good ones!) or most ordinary conversations. Yet there are very few records of actual, mind-changing conversations online, despite their importance in the rationality community.
Posting things publicly online increases the surface area of ideas to the people who might use them, and can have very positive, hard-to-predict effects.

Introduction

Facilitator: One way to start would be to get a bit of both of your senses of the importance of forecasting, maybe Ozzie starting first. Why are you excited about it and what caused you to get involved?

Ozzie: Actually, would it be possible that you start first? Because there are just so many ...

Vaniver: Yeah. My sense is that predicting the future is great. Forecasting is one way to do this. The question for “will this connect to things being better” is the difficult part. In particular, Ozzie had this picture before of, on the one hand, data science-y repeated things that happen a lot, and then on the other hand judgement-style forecasting, a one off thing where people are relying on whatever models because they can't do the “predict the weather”-style things.

Vaniver: My sense is that most of the things that ...
Does improved introspection cause rationalisation to become less noticeable?
by Bird Concept — 2019-07-30T10:03:00.202Z

I've recently updated that noticing is a key rationality skill -- not just noticing confusion, but noticing your cognition more generally. This allows you to figure out at a very granular level why you're not reaching your goals, and then intervene to change those reasons.

For example:

At one point I found myself procrastinating on ordering the catering for an event. Noticing the disconnect between my high-level goals ("make a good event") and my concrete actions ("spend time on FB"), triggered me to try to notice what was up in my mind (this is a particular trigger-response pattern I've trained myself to use). I found that I didn't want to make the call since last time I called them, they couldn't hear what I was saying and were kind of rude about it. I didn't want my phone to be bad or my accent to be inaudible, an so I didn't want to call them again. I then proceeded to borrow a friend's phone, and called them without problem.

Another example, this time with a concrete cognitive rather than practical intervention:

I noticed myself being unhappier than I wanted to be. So when the unhappiness clashed with the higher-level desire for happiness, it triggered a noticing process, and I realised my mind was running an algorithm like: "notice happy thought --> remember Hamming problem or that timelines might be short --> feel bad". This sounds ridicolously unhelpful when written out, but is in fact what was going on. So I started training myself to hold on to the happiness in the first part of the chain without automatically falling into the second.

Here's a worry with this: if part of my congition is consciously accessible and interpretable, and part of it is not, will extensive noticing-and-intervening cause motivated cognition to become less noticable?

It will by selection effects, since the more noticeable parts I'll change. But this feels more definitionally true than actually worrying.

It also might by negative reinforcement, if my mind learns that when subagents make their desires known they'll tend to be overruled/modified. (To prevent this, and as a safer policy in my current epistemic state, I make sure to sometimes deliberately not intervene on things I've noticed.) But this shouldn't be the case if I genuinely listen to subagents and take their preferen...
On Framing Political Opinions to Quickly Assess the Crux of Disagreement
by Tuk — 2019-07-30T03:07:00.515Z

The following will be elementary to many of you, and for that I must explain that this idea popped into my head fully formed and I need to improve my writing habit, so here goes. Hopefully this will be insightful for someone!

Political opinions can be framed as a set of three components: Cause, Means, and Ends (consequences).

Example: Poor people can’t afford rent in this town on minimum wage, so you should vote for a Democrat to enact higher minimum wage laws.

Let’s unpack this as follows:

Cause: Poor people can’t afford rent in this town on minimum wage

Means: vote for a Democrat to enact higher minimum wage laws

Ends: Unstated! But we can infer that the claim is that higher minimum wage laws create a world in which poor people can afford rent in this town.

I used this example specifically because it’s a common argument, and the ends are unclear. I have noticed that the majority of political movements and ideologies have very vague Ends. This is in part due to how difficult it is to predict the future and the outcomes of different government policies. But part of it is to prevent disagreements from occurring regarding the Ends.

So you may be in an argument with someone who agrees with you about the Cause, agrees about the Ends, but disagrees about the Means, and any other combination.

I’ve been in political movements where everyone allied with a Cause, but then it fell apart from infighting about Means, and we barely even touched Ends! It’s very rare that we meet people who match our beliefs on all three components.

Another element of this is that our minds create meaning by finding patterns and matching those patterns to categories which we then label. We use those labels often without realizing how fuzzy the borders of our categories are, and then sometimes we argue about labels instead of realizing that we’re arguing about where the boundaries of the categories are, or whether a Thing belongs in the category or not.

Here are some examples to illustrate what I mean. When I was part of Occupy Wall Street, we surveyed the attendees to determine what causes people agreed with. We found a huge array of different causes brought us together, but a few stood out as central issues. Then we attempted to discuss solutions and quickly realized how many different worldviews were held by our group members. Eventually, the fundamental disagreements were insurmountable, and the ...
Information empathy
by Sunny from QAD — 2019-07-30T01:32:45.174Z

Epistemic status: Giving a name to a basic skill that some people exhibit and others don't, because I think that it's useful to keep track of who exhibits it and that giving it a name will help with this. See also: Historian's Fallacy, Theory of Mind (both Wikipedia links).

Edit: Down below, I was convinced that this is not a useful term because it is nearly identical in meaning to Theory of Mind, and that the latter can be substituted in whenever you would use this term. This post might still be worth reading if you don't know what that is, but please read mentally substitute the official phrasing for my coinage.

The experiment I'm about to describe may be apocryphal (links, anyone?), but it illustrates my point nicely. (Edit: down below, Raemon has linked the real experiment, which is slightly different but has exactly the same moral.)

In the experiment, a child is shown a box with a "trick" lid, such that when the lid is on the box it appears that there is a cookie inside, but when the lid is removed you can see that it is in fact empty. First, with the lid on the box, the child is asked what is inside. "A cookie," they answer. Then the lid is removed, and again the child is asked what is inside. "Nothing," they say. Finally, the lid is replaced, and the experimenter tells the child that they are going to show the box to another child, who has not yet seen the box with the lid off.

"What will the other child think is in the box?" the experimenter asks.

At this point, things can go one of two ways. Children beyond a certain developmental stage will say "she'll say there's a cookie", while children who have not yet reached this stage will say "she'll say it's empty".

I call this skill Information Empathy, and from what I can tell, the median age at which children develop it is 35.

Jokes aside, I feel like there have been actual occasions where somebody has been angry at me after I did X, where the primary reason for their anger seems to be neither that X happened to them nor that my doing X was careless negligent but rather that I did X knowing full well what the consequences were... even when they could see that that was not true.

Maybe this term will be useful for some of you.
Does it become easier, or harder, for the world to coordinate around not building AGI as time goes on?
by Eli Tyre — 2019-07-29T22:59:33.170Z

(Or, is coordination easier in a long timeline?)

It seems like it would be good if the world could coordinate to not build AGI. That is, at some point in the future, when some number of teams will have the technical ability to build and deploy and AGI, but they all agree to voluntarily delay (perhaps on penalty of sanctions) until they’re confident that humanity knows how to align such a system.

Currently, this kind of coordination seems like a pretty implausible state of affairs. But I want to know if it seems like it becomes more or less plausible as time passes.

The following is my initial thinking in this area. I don’t know the relative importance of the factors that I listed, and there’s lots that I don’t understand about each of them. I would be glad for…

Additional relevant factors.
Arguments that some factor is much more important than the others.
Corrections, clarifications, or counterarguments to any of this.
Other answers to the question, that ignore my thoughts entirely.

If coordination gets harder overtime, that’s probably because...

Compute increases make developing and/or running an AGI cheaper. The most obvious consideration is that the cost of computing falls each year. If one of the bottlenecks for an AGI project is having large amounts of compute, then “having access to sufficient compute” is a gatekeeper criterion on who can build AGI. As the cost of computing continues to fall, more groups will be able to run AGI projects. The more people who can build an AGI, the harder it becomes to coordinate all of them into not deploying it.

Note that It is unclear to what degree there is currently, or will be, a hardware overhang. If someone in 2019 could already run an AGI, on only $10,000 worth of AWS, if only they knew how, then the cost of compute is not relevant to the question of coordination.

The number of relevant actors increases. If someone builds an AGI in the next year, I am reasonably confident that that someone will be Deep Mind. I expect that in 15 years, if I knew that AGI would be developed one year from then, it will be much less overdetermined which group is going to build it, because there will be many more well funded AI teams with top talent, and, most likely, none of them will have as strong a lead as Deep Mind currently appears to have.

This consideration suggests that coordination gets harder over time. However, this depends heavily

...
Toy model piece #1: Partial preferences revisited
by Stuart_Armstrong — 2019-07-29T16:35:19.561Z

EDIT: This model is currently obsolete, see here for the most current version.

I'm working towards a toy model that will illustrate all the steps in the research agenda. It will start with some algorithmic stand-in for the "human", and proceed to create the UH, following all the steps in that research agenda. So I'll be posting a series of "toy model pieces", that will then be ultimately combined in a full toy model. Along the way, I hope to get a better understanding of how to do the research agenda in practice, and maybe even modify that agenda based on insights making the toy model.

In this post, I'll revisit and re-formalise partial preferences, and then transform them into utility functions.

The problem with the old definition

My previous model of partial preferences can't capture some very simple mental models, such as P="the more people smile, the better the world is".

This is because the partial preference decomposes the space of worlds locally as Y×Z, fixes two values y− and y+ in Y, and only compares worlds of type (y−,z) and (y+,z) for fixed z. This means that we can only compare worlds with the same z value, and only two of these worlds can be compared: so we can't say w1<w2<w3 for three distinct worlds. Thus we can't say that three people smiling is better than two, which is better than one. Not being able to capture preferences like P is a major flaw.

New definition: preorder

So now model a partial preference as preorder. A preorder ≤ is a type of ordering that is transitive (if w1≤w2 and w2≤w3, then w1≤w3) and reflexive (w≤w for all worlds w).

The previous type of partial preference can be made into a preorder quite easily: w1<w2 implies w1≤w2, and add w≤w for all worlds w.

Now we can easily express P="the more people smile, the better the world is". Let w(n,v) be a world with n smiling people in it, with v representing all the other relevant variables describing the world. The P is described by the preorder:

w(n,v)≤w(n,v′) if and only if v=v′ and n≤m.

For a general preorder ≤, define w<w′ to mean w≤w′ but it not being the case that w′≤w.

Circular preferences and utility functions

Unfortunately, unlike the previous partial preferences, preorders can allow for circular preferences w1≤w2≤w3≤w1. In practice, most sensible partial preferences will not have circular preferences, and will instead resemble P: just a collection of orderings a

...
Mapping of enneagram to MTG personality types
by Alexei — 2019-07-29T15:20:09.115Z

For fun, Parina and I mapped enneagram types to a pair of Magic: The Gathering colors. (Following in the great tradition of https://medium.com/s/story/the-mtg-color-wheel-c9700a7cf36d) For each color pair, there’s no ordering; think of it as a set.

Without further ado:

1 Reformer
White red

2 Helper
Black green

3 Achiever
Black white

4 Individualist
Blue green

5 Investigator
Blue red

6 Loyalist
Black blue / blue white

7 Enthusiast
Red green

8 Challenger
Black red

9 Peacemaker
Green white

Let me know if this helps or resonates with you.
What woo to read? 
by calebo — 2019-07-29T15:19:08.788Z

Recently, I've heard more discussion of chakras, energy, and similar ideas in rationalist type and adjacent circles.

I don't really know where this comes from. So I thought I'd ask, what "woo" do you all find useful?

I'm interested in:

pointers to explanations of woo

explicit explanations of why woo is useful for rationalist types
Is there neuroscience research on cognitive biases?
by Yoav Ravid — 2019-07-29T07:45:04.365Z

I recently watched NeuraLink's presentation, and wondered how can something like that help us reason.

The obvious way is an AI that can reason, and is connected to our brains and we reason together with it.

But another direction i thought of, is just helping us notice when we're using motivated cognition and letting cognitive biases take place.

Another thing i thought of was reducing Akrasia, can it help us win the fight between areas in the brain in an akratic situation?

With my very little knowledge about the subjects in hand, it seems like it would be an easier target - seems to me you'd need fewer electrodes, less understanding of how reasoning works, and simpler software.

Although the questions about the possibilities of this technology are interesting, i don't expect anything more the guesses and predictions to be available right now.

So my question is whether we have neurological knowledge about how these mechanisms work.
Compilers/PLs book recommendation?
by johnswentworth — 2019-07-28T15:49:17.570Z

I'm interested in:

Data structures for representing programs
Algorithms for translating between those representations
Analysis & manipulations of each representation; questions which are natural to ask/answer in each

Most books I can find on compilers/PLs tend to spend most of their time on the text representation (and algorithms for translating programs out of text, i.e. parsing) and the machine-code representation (and algorithms for translating programs into machine code). For purposes of this question, I'm not particularly interested in either of these representations - they're not very natural data structures for representing programs, and we mostly use them because we have to.

I'm interested mainly in high-level "intermediate" representations. In terms of applications, I'm more interested in reasoning about about the code (e.g. control flow analysis, type checking) and manipulating the code (e.g. automated parallelization, high-level machine-independent optimization methods) rather than translating between any particular source/target format.

Questions:

Does anybody know of a good book (or other source) that focuses on high-level "intermediate" representations of programs?
Is there some other question I should be asking, e.g. a different term to search for?
On the meta-level, where else should I look/ask this question?